Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications

Even with the increase in the number and variety of computer resources available to research scientists today, it is still challenging to construct scalable distributed applications. To address this issue, we developed Work Queue, a flexible master/worker framework for building large scale scientific ensemble applications that span many machines including clusters, grids, and clouds. In this paper, we describe Work Queue and then present the Python-WorkQueue module, which enables scientists to take advantage of our Work Queue framework while using the Python programming language. To demonstrate the module’s flexibility and power, we examine two distributed scientific applications, RepExWQ and Folding@work. Both of these programs were written using Python-WorkQueue and manifest the Work Queue framework’s ability to scale not only to hundreds of workers, but to also enable scientists to take advantage of multiple distributed computing resources simultaneously. Keywords-python; work queue; master/worker; distributed systems; cloud computing; molecular dynamics